Reduced - error pruning with signi cance testsEIBE

When building classiication models, it is common practice to prune them to counter spurious eeects of the training data: this often improves performance and reduces model size. \Reduced-error pruning" is a fast pruning procedure for decision trees that is known to produce small and accurate trees. Apart from the data from which the tree is grown, it uses an independent \pruning" set, and pruning decisions are based on the model's error rate on this fresh data. Recently it has been observed that reduced-error pruning overrts the pruning data, producing unnecessarily large decision trees. This paper investigates whether standard statistical signiicance tests can be used to counter this phenomenon. The problem of overrtting to the pruning set highlights the need for signiicance testing. We investigate two classes of test, \parametric" and \non-parametric." The standard chi-squared statistic can be used both in a parametric test and as the basis for a non-parametric permutation test. In both cases it is necessary to select the signiicance level at which pruning is applied. We show empirically that both versions of the chi-squared test perform equally well if their signiicance levels are adjusted appropriately. Using a collection of standard datasets, we show that signiicance testing improves on standard reduced error pruning if the signiicance level is tailored to the particular dataset at hand using cross-validation, yielding consistently smaller trees that perform at least as well and sometimes better.